SRE-3150 ci: enable build and HW Tests on el9.7 #17327

grom72 · 2025-12-30T15:19:09Z

This PR update Build and HW Medium Tests stages in CI to use el9.7 instead of el8.8.

The new version of the post_provisioning_config_nodes.sh script introduces a new approach to setting up NVME disks.

This approach assumes that the kernel randomly enumerates NVMe devices during boot time.
It uses PCIe addresses rather than logical device names (such as /dev/nvmeX) to set up the correct NVME disk configuration.
It reserved the two first NVME devices per NUMA (ordered by PCIe addresses) for DAOS and vfio-pci driver
Dummy file systems have been created on all other disks to prevent DAOS/SPDK from using them.
The procedure assumes that any NVMe disks that are already mounted are required by the OS and should not be used by the DAOS test environment (e.g. NVMe disks used for OS booting).

**

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: ravalsam <[email protected]>

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: ravalsam <[email protected]> Signed-off-by: Samir Raval <[email protected]>

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: ravalsam <[email protected]>

Skip-build-el9-gcc: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Skip-build-el9-gcc: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Signed-off-by: Tomasz Gromadzki <[email protected]>

This reverts commit e3da5b4. Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw

Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Doc-only: true Signed-off-by: Tomasz Gromadzki <[email protected]>

Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Skip-test-rpms: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Signed-off-by: Tomasz Gromadzki <[email protected]>

Let's try to run fix with original configuration to confirm that problem still exists. Cancel-prev-build: false Quick-build: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-test: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Skip-test-rpms: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Quick-build: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-test: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Skip-test-rpms: true Test-tag: test_recycle_pools_hw Revert "Run Tests without fix to collect more data" This reverts commit 2bae9ce. Signed-off-by: Tomasz Gromadzki <[email protected]>

clamav must be removed as required dependency is not avaialble: "nothing provides libcrypto.so.3(OPENSSL_3.4.0)(64bit)" Cancel-prev-build: false Quick-build: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-test: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Skip-test-rpms: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Quick-build: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-test: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Skip-test-rpms: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

This reverts commit e2ba246. Cancel-prev-build: false Quick-build: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-test: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Skip-test-rpms: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Cancel-prev-build: false Quick-build: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-test: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Skip-test-rpms: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

daosbuild3 · 2026-01-02T22:03:41Z

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/19/execution/node/696/log

daosbuild3 · 2026-01-02T22:34:20Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/19/execution/node/856/log

Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: true Skip-unit-test-memcheck: true Allow-unstable-test: true Skip-func-vm: true Skip-func-test-vm-valgrind: true Skip-func-test-el8: true Skip-func-test-el9: true Skip-func-test-leap15: true Skip-fault-injection-test: true Skip-test-el-8.6-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test-medium: false Skip-func-hw-test-medium-vmd: false Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-medium-verbs-provider: false Skip-func-hw-test-medium-ucx-provider: false Skip-func-hw-test-large: false Skip-func-hw-test-large-md-on-ssd: false Signed-off-by: Tomasz Gromadzki <[email protected]>

daosbuild3 · 2026-01-02T23:23:00Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/20/display/redirect

daosbuild3 · 2026-01-03T00:09:52Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/21/execution/node/670/log

daosbuild3 · 2026-01-03T01:34:48Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/19/execution/node/681/log

daosbuild3 · 2026-01-03T04:10:11Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/21/execution/node/660/log

JohnMalmberg · 2026-01-09T14:35:50Z

ci/provisioning/post_provision_config_nodes_EL.sh

    dnf -y reinstall sssd-common
+    # Seems to be required until https://daosio.atlassian.net/browse/DAOS-18358
+    # is fixed.
+    dnf -y remove clamav-lib


This is because el-9.6 is out of standard support and clamav-lib is no longer supports the out of support release. It should work with the current el-9.7 release.

Signed-off-by: Tomasz Gromadzki <[email protected]>

daosbuild3 · 2026-01-12T11:13:32Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/24/display/redirect

Cancel-prev-build: false Priority: 2 Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests:true Skip-unit-test: true Skip-NLT: true Skip-unit-test-memcheck: true Allow-unstable-test: true Skip-func-vm: true Skip-func-test-vm-valgrind: true Skip-func-test-el8: true Skip-func-test-el9: true Skip-func-test-leap15: true Skip-fault-injection-test: true Skip-test-el-8.6-rpms: true Skip-test-leap-15-rpms: true Skip-func-hw-test-medium: false Skip-func-hw-test-medium-vmd: false Skip-func-hw-test-medium-md-on-ssd: false Skip-func-hw-test-medium-verbs-provider: false Skip-func-hw-test-medium-ucx-provider: false Skip-func-hw-test-large: false Skip-func-hw-test-large-md-on-ssd: false Signed-off-by: Tomasz Gromadzki <[email protected]>

daosbuild3 · 2026-01-12T13:15:51Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/24/execution/node/669/log

daosbuild3 · 2026-01-12T13:41:58Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/25/display/redirect

daosbuild3 · 2026-01-12T13:42:37Z

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/25/display/redirect

daosbuild3 · 2026-01-12T13:52:19Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/25/display/redirect

daosbuild3 · 2026-01-12T13:52:26Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/25/display/redirect

daosbuild3 · 2026-01-12T13:59:04Z

Test stage Functional Hardware Medium completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/25/display/redirect

daosbuild3 · 2026-01-12T14:01:25Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/25/display/redirect

daosbuild3 · 2026-01-12T14:01:30Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/25/display/redirect

daosbuild3 · 2026-01-12T14:10:28Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/25/execution/node/890/log

daosbuild3 · 2026-01-12T20:17:20Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/26/display/redirect

daosbuild3 · 2026-01-12T20:17:25Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/26/display/redirect

daosbuild3 · 2026-01-12T20:21:47Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/26/display/redirect

daosbuild3 · 2026-01-12T20:21:54Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/26/display/redirect

daosbuild3 · 2026-01-12T20:23:46Z

Test stage Functional Hardware Medium Verbs Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/26/display/redirect

daosbuild3 · 2026-01-12T20:24:22Z

Test stage Functional Hardware Large completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/26/execution/node/725/log

daosbuild3 · 2026-01-12T20:28:13Z

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-17327/26/display/redirect

daosbuild3 · 2026-01-13T18:06:31Z

Test stage Functional Hardware Medium UCX Provider completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17327/27/execution/node/805/log

ravalsam and others added 30 commits December 30, 2025 15:07

SRE-3150 Test: This is Debug PR for NVMe issue

d5cb8e0

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: ravalsam <[email protected]>

Reverting it back to original OS.

6c78311

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: ravalsam <[email protected]>

Comment node prep NVMe check.

fdb99d2

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: ravalsam <[email protected]>

Update test_main_prep_node.sh

9f60be9

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: ravalsam <[email protected]> Signed-off-by: Samir Raval <[email protected]>

Testing changes.

e4a7c38

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: ravalsam <[email protected]>

Break tests repetition after first failure

3f3b111

Skip-build-el9-gcc: true Skip-build-leap15-gcc: true Skip-build-leap15-icc: true Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Sanity check on el9.5

19a6a3c

Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Skip leap 15 build

8aed634

Signed-off-by: Tomasz Gromadzki <[email protected]>

Revert "Sanity check on el9.5"

481de85

This reverts commit e3da5b4. Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

Let's try 9.5 again

48bc314

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw

Add parameter to controll HW stages image version

74ebd21

Signed-off-by: Tomasz Gromadzki <[email protected]> Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw

Stop tests as soon as fault is detected

455f4d5

Cancel-prev-build: false Skip-unit-tests: true Skip-func-vm: true Skip-fault-injection-test: true Test-tag: test_recycle_pools_hw Signed-off-by: Tomasz Gromadzki <[email protected]>

nvme drives setup script

fde1309

Doc-only: true Signed-off-by: Tomasz Gromadzki <[email protected]>

Increase max repetition number for cluster sanity check

5b7b932

Signed-off-by: Tomasz Gromadzki <[email protected]>

Update script to enable two devices per numa node

a77ad8b

Signed-off-by: Tomasz Gromadzki <[email protected]>

Speed up test by les repetitions

0ad5d6e

Signed-off-by: Tomasz Gromadzki <[email protected]>

Avoid error in case of unbinding failure

504ad56

Signed-off-by: Tomasz Gromadzki <[email protected]>

JohnMalmberg reviewed Jan 9, 2026

View reviewed changes

Update to el9.7

91dc31d

Signed-off-by: Tomasz Gromadzki <[email protected]>

grom72 changed the title ~~SRE-3150 ci: enable HW Tests on el9.6~~ SRE-3150 ci: enable build and HW Tests on el9.7 Jan 12, 2026

grom72 force-pushed the grom72/SRE-3150 branch from 84727fe to b40f7a8 Compare January 12, 2026 11:59

SRE-3150 ci: enable build and HW Tests on el9.7 #17327

Are you sure you want to change the base?

SRE-3150 ci: enable build and HW Tests on el9.7 #17327

Conversation

grom72 commented Dec 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 2, 2026

Uh oh!

daosbuild3 commented Jan 3, 2026

Uh oh!

daosbuild3 commented Jan 3, 2026

Uh oh!

daosbuild3 commented Jan 3, 2026

Uh oh!

JohnMalmberg Jan 9, 2026

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 12, 2026

Uh oh!

daosbuild3 commented Jan 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

grom72 commented Dec 30, 2025 •

edited

Loading